Image experiencing system and information processing method

ABSTRACT

The invention intends to increase the feeling of reality in a board game in addition to the interestingness of the game itself and to facilitate understanding of the proceeding situation of the game.  
     The invention provides an image experiencing system for a game which proceeds by placing items on a game board, comprising player position/attitude determining unit for determining the position/direction information of the view of the player, a generation unit for generating computer graphics based on the items on the game board, according to the position/attitude information of the view of the player, and a head-mounted display for displaying the generated computer graphics in superposition with the image of the real field.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a mixed reality technology forproviding a game which proceeds by positioning items on a game board.

[0003] 2. Related Background Art

[0004] Among various games, there is already known a board game, whichutilizes a board including areas divided thereon and proceeds byplacing, removing or displacing pieces on such areas. Among the gamesutilizing objects of certain three-dimensional shapes as such pieces,there are well known, for example, chess, checker, backgammon, igo,Japanese chess, Japanese backgammon etc. Also there are known gamesutilizing cards as such pieces, called card games. The games utilizingplaying cards are known in numerous kinds, such as bridge, stud poker,draw poker, black jack etc.

[0005] Also among the card games, there is known so-called card battle,which utilizes cards specific to the game and in which each card isgiven a specific function. Also many games utilizing the playing cardsdo not use a particular board because the divided areas are quitesimple, but, in such games, it can be considered that a board includinginvisible divided areas is present and its existence is recognized andshared by the players.

[0006] Such board game or card game itself often assumes a certainevent, and the item (for example piece) often assumes a particularanimal or a particular person. The board or piece has a shape determinedin advance, and the pattern thereof does not change according to theproceeding of the game.

[0007] On the other hand, there is also known a game utilizing the MR(mixed reality) technology. In such game, the environment of the game isconstructed with a real setting with scene settings and stageproperties, and the players execute the game by actually entering suchenvironment. In most cases, each player wears a see-through HMD (headmounted display), which displays a CG (computer graphics) image matchingthe proceeding of the game, in superposition with an image that can beseen when the HMD is not worn.

[0008] In the conventional board games mentioned above, the shape orpattern of the pieces do not change according to the situation of thegame. For example, in a battle scene, it is not that an actual battletakes place in front of the player, or, in case the player draws a cardindicating “the angel gives an instruction”, it is not that an angleactually speaks up.

[0009] Therefore, even though the game itself assumes a certain scene,the game lacks the feeling of reality because of the lack ofcorresponding display. Also for a similar reason, it is difficult tograsp the situation of proceeding of the game at a sight.

[0010] On the other hand, the conventional MR game mentioned in theforegoing provides sufficient feeling of reality but involves a verytedious setting of the game environment. There is often required alarge-scale work for preparing the scene setting, and the positions ofthe objects in the setting have to be measured for each setting. It isalso difficult to alter the content of the game.

SUMMARY OF THE INVENTION

[0011] In consideration of the foregoing, an object of the presentinvention is to improve the feeling of reality of a board game inaddition to the interestingness thereof, and to facilitate understandingof the situation of proceeding of the game.

[0012] Another object of the present invention is, in comparison withthe conventional games utilizing the MR technology, to facilitateinstallation of the setting and to enable relatively flexible alterationof the content of the game.

[0013] The above-mentioned objects can be attained, according to thepresent invention, by an image experiencing system for a game whichproceeds by placing items on a game board, the system comprising:

[0014] player position and attitude determining means for obtainingposition/attitude information of the view of a player;

[0015] generation means for generating computer graphics according tothe items of the game board, corresponding to the position/attitudeinformation of the view of the aforementioned player; and

[0016] a head mounted display capable of displaying thus generatedcomputer graphics in superposition with the image of a real world.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017]FIG. 1 is a view showing an example of the configuration of animage experiencing system in a first embodiment;

[0018]FIG. 2 is a view showing a board and the appearance thereof incalculating the height of view point of a player;

[0019]FIG. 3 is a view showing the internal configuration of asee-through HMD;

[0020]FIG. 4 is a view showing an example of the configuration of animage experiencing system in a first embodiment;

[0021]FIG. 5 is a view showing an example of the configuration of animage experiencing system in a second embodiment;

[0022]FIG. 6 is a view showing an example of the configuration of animage experiencing system in a third embodiment;

[0023]FIG. 7 is a UML activity chart showing the process of a mostlikelihood position and attitude determining unit constituting acomponent of the image experiencing system of the third embodiment;

[0024]FIG. 8 is a chart indicating the difference between a value,measured with a position and attitude sensor constituting a component ofthe image experiencing system of a fourth embodiment, and a real valueas a function of elapsing time;

[0025]FIG. 9 is a view showing an example of the configuration of animage experiencing system in a fourth embodiment;

[0026]FIG. 10 is a UML activity chart showing the process of aposition-and-attitude sensor information processing unit constituting acomponent of the image experiencing system of the fourth embodiment;

[0027]FIG. 11 is a view showing an example of the configuration of animage experiencing system in a fifth embodiment;

[0028]FIG. 12 is a UML activity chart showing the process of a mostlikelihood position and attitude determining unit constituting acomponent of the image experiencing system of the fifth embodiment;

[0029]FIG. 13 is a view showing an example of the configuration of animage experiencing system in a sixth embodiment;

[0030]FIG. 14 is a UML activity chart showing the process of an attitudesensor information processing unit constituting a component of the imageexperiencing system of the sixth embodiment;

[0031]FIG. 15 is a view showing an example of the configuration of animage experiencing system in a seventh embodiment;

[0032]FIG. 16 is a UML activity chart showing the process of a pieceoperation recognition unit constituting a component of the imageexperiencing system of the seventh embodiment;

[0033]FIG. 17 is a view showing an example of the configuration of animage experiencing system in a tenth embodiment;

[0034]FIG. 18 is a view showing an example of card patterns to be usedfor explaining the function of the image experiencing system of thetenth embodiment;

[0035]FIG. 19 is a view showing recognition areas on a card, to be usedfor explaining the function of the image experiencing system of thetenth embodiment;

[0036]FIG. 20 is a UML activity chart showing the process of a pieceimage recognition unit constituting a component of the imageexperiencing system of the tenth embodiment;

[0037]FIG. 21 is a view showing an example of the configuration of animage experiencing system in a eleventh embodiment;

[0038]FIG. 22 is a UML activity chart showing the process of an on-boardpiece image recognition unit constituting a component of the imageexperiencing system of the eleventh embodiment;

[0039]FIG. 23 is a view showing an example of the configuration of animage experiencing system in a twelfth embodiment;

[0040]FIGS. 24 and 25 are UML activity charts showing the process of apiece operation recognition unit constituting a component of the imageexperiencing system of the twelfth embodiment;

[0041]FIG. 26 is a view showing an example of the configuration of animage experiencing system in a thirteenth embodiment;

[0042]FIG. 27 is a view showing the difference in the output from thepiece image recognition unit, for explaining a piece imagerecognition-guide display instruction unit constituting a component ofthe image experiencing system of the thirteenth embodiment;

[0043]FIG. 28 is a UML activity chart showing the process of the pieceimage recognition-guide display instruction unit constituting acomponent of the image experiencing system of the thirteenth embodiment;

[0044]FIG. 29 is a view showing an example of the guide to be displayedin the display unit of the HMD by the image experiencing system of thethirteenth embodiment;

[0045]FIG. 30, composed of FIGS. 30A and 30B, and FIG. 31, composed ofFIGS. 31A, 31B and 31C, are views showing examples of the configurationof an image experiencing system in a fourteenth embodiment;

[0046]FIG. 32 is a view showing a fifteenth embodiment in a conceptualimage;

[0047]FIG. 33 is a schematic view showing the configuration of thefifteenth embodiment;

[0048]FIG. 34 is a view showing markers;

[0049]FIG. 35 is a view showing guides;

[0050]FIG. 36 is a view showing card identification;

[0051]FIG. 37 is a view showing a phase proceeding by voice;

[0052]FIG. 38 is a flow chart of a card reading unit;

[0053]FIG. 39 is a flow chart of a position and attitude grasp unit; and

[0054]FIG. 40 is a flow chart of a game management unit.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] Now the present invention will be clarified in detail by thefollowing description, which is to be taken in conjunction with theaccompanying drawings, in which equivalent configurations arerepresented by same numbers.

[0056] In the image experiencing system to be explained in thefollowing, each player wears a see-through HMD (head mounted display)for displaying CF (computer graphics) in superposition with the actualsituation of the board game or card game, executed in a limited field ofa game board. The CG changes according to the proceeding of the game.For example, in a chess game, a knight piece is represented by CG of aknight on horse back, and at the displacement of the piece, the CG showsa running horse. Also, when a piece captures an opponent piece, there isgiven fighting and winning CG against CG corresponding to the opponentpiece.

[0057] Such image experiencing system provides, in addition to theinterestingness of the actual board game itself, an improved feeling ofreality and facilitates grasping the situation of proceeding of thegame.

[0058] Also in comparison with the conventional game utilizing the MRtechnology, the image experiencing system of the present invention iseasier in setting and allows to relatively easily accommodate alterationof the content of the game.

[0059] (First Embodiment)

[0060]FIG. 1 is a view showing an example of the configuration of theimage experiencing system of a first embodiment.

[0061] There is provided a game board 101 constituting the field ofgame, and players execute the game by placing, removing or moving pieceson the board 101. The player wears a see-through HMD 103 in playing thegame. A position and attitude sensor 104 is fixed to the HMD 103 anddetects the position and attitude of the view of the player.

[0062] In the following there will be given definitions for the terms“position/attitude (posture)”, “position” and “attitude” to be used inthe present specification. “Position/attitude” includes both the“position” and “attitude”. Thus, “position/attitude information” meansboth the “position information” and “attitude information”.

[0063] “Position” means information indicating a point in a specifiedspatial coordinate system, and is represented, in case of an XYZorthogonal coordinate system, by a set of three values (x, y, z). Alsoin case of representing an object on the earth, there can be employed aset of three values of a latitude, a longitude and a height (or adepth). “Attitude” means a direction from the point represented by the“position”, and can be represented by the position of an arbitrary pointon such direction, or, in case of the XYZ orthogonal coordinate system,by the angles of the viewing line with the axes of the coordinate systemor by specifying a direction of the viewing line (for example -Zdirection) and indicating the amounts of rotation from such specifieddirection about the axes of the coordinate system.

[0064] In the absence of other limiting conditions, the “position” has 3freedoms, and the “attitude” also has 3 freedoms.

[0065] The position and attitude sensor is capable of values of 6freedoms on “position” and “attitude”.

[0066] Such position and attitude sensor is commercially available invarious forms, for example one utilizing a magnetic field, one based onimage processing of a marker photographed by an external camera, or onebased on the combination of a gyro sensor and an acceleration sensor.

[0067] Because of the nature of the board game, the head position of theplayer scarcely changes during the play. It is therefore possible alsoto use the position information calibrated at the start of the game andto use an attitude sensor which measures the attitude information only.In such case, the measurement is made only on the attitude informationof 3 freedoms during the game, but the system can process the values of6 freedoms including the position information initially calibrated. Inother words, the attitude sensor and the calibrated position data can beconsidered to constitute a position and attitude sensor.

[0068]FIG. 2 shows a method of calibration before the game. A game board101 is provided with markers 201 for identification on four corners. Forthe purpose of simplicity, it is assumed that the markers 201 areprovided in a square arrangement, with a length of a side of unity (1).The player is positioned in front of the board, with a view point 202positioned at the center position of the board and at a distance d fromthe front side of the board. When the player observes the board in thisstate, the rear side of the board appears shorter than the front side.Based on an observed length m₁ of the front side and an observed lengthm₂ of the rear side, the height h of the view point 202 from the planeof the board 101, though dependent on the projection method, can bedetermined as:

h=((m ₂ ²(d+1)² m ₁ ² d ²)/(m ² m ₂ ²))^(0.5)

[0069] The input/output of the image to or from the HMD 103 and theposition/attitude information from the position/attitude sensor 104 areprocessed by a game console or a PC 102.

[0070]FIG. 3 is a view showing the internal configuration of thesee-through HMD 103, which includes a video see-through type and anoptical see-through type.

[0071] In case of the video see-through type, the light from theexternal field does not directly reach the eyes of the player. The lightfrom the external field is deflected by a two-sided mirror 301 andenters an image pickup device 302. An image presented to the player isdisplayed on a display device (display unit) 303 and enters the eyes theplayer via the two-sided mirror 301. If the output image of the imagepickup device (image pickup unit) 392 is directly supplied to thedisplay device 303, the HMD becomes a mere seeing glass, but such outputimage is processed by the game console or PC 102 in the course of suchsupply to display the generated CG in superposition.

[0072] In case of the optical see-through type, the light directlyreaches the eyes of the player, and the separately generated CG aresimultaneously displayed and appear in superposition to the eyes of theplayer.

[0073] The light from the external field pass through a half mirror andenters the eyes of the player. At the same time, an image displayed on adisplay device is reflected by the half mirror and enters the eyes ofthe player. The image pickup device is unnecessary in this case, but itis required if an image at the view point of the player is used in imageprocessing. Instead of the image pickup device 302, a separate camerafor image processing may be fixed on the HMD 103.

[0074] The game console or PC 102 manages the proceeding of the game asin the ordinary game.

[0075] In case of a board game, the required position/attitudeinformation is limited to the position/attitude relationship between theboard 101 and the HMD 103, and there is not required setting of thescene or stage properties at each installation or the calibration of thesensors to be mounted on the players.

[0076] The present embodiment requires only a more compact set, incomparison with the MR game, and is easy in the installation work,including the calibration. The details of the MR game are described, forexample, in the 22A: Mixed Reality in the papers of the fourthConvention of the Japanese Virtual Reality Society, by “Design andImplementation for MR Amusement Systems”.

[0077] The present embodiment can also improve the feeling of reality incomparison with the conventional board games.

[0078]FIG. 4 shows an example of the configuration of an imageexperiencing system in which the present embodiment is applied.

[0079] The game console or PC 102 manages the proceeding of the game bya game management unit 401. A CG generation unit 405 generates CG(computer graphics) corresponding to each scene. For generating a CGimage seen from the view point of the player, the CG generation unit 405acquires the position and attitude information of the view of the playerfrom a player position and attitude determining means 402, whichincludes, for example, a position and attitude sensor 104, and aposition/attitude sensor information processing unit 403 for analyzingsuch information thereby determining the position and attitudeinformation of the view of the player.

[0080] The position and attitude sensor information processing unit 403executes, for example, a format conversion of the data obtained from theposition and attitude sensor 104, a transformation into a coordinatesystem employed in the present system, and a correction for thedifference between the mounting position of the position and attitudesensor and the view point of the HMD 103.

[0081] The CG, generated in the CG generation unit 405 and correspondingto an image seen from the view point of the player, are superimposed inan image composition unit 404, in case of the HMDof video tYsee-throughtype, with an image obtained from the image pickup unit 302 of the HMD103, for display on the image display unit 303.

[0082] In case of the HMD of optical see-through type, the image pickupunit 302 and the image composition unit 404 can be dispensed with sincethe image synthesis is unnecessary, and the output of the CG generationunit 405 is directly displayed on the display unit 303.

[0083] The game management unit 401 stores information relating to thegame itself, or the rules of the game, and, in the course of a game,retains the current status or scene and determines and manages a nextstate to which the game is to proceed. Also for presenting a scene by CGto the player, it issues a drawing instruction for CG to the CGgeneration unit 405.

[0084] According to the instruction from the game management unit 401,the CG generation unit 405 places model data, which are an internalrepresentation corresponding to each character, in a world, which is aninternal representation of a virtual world in which the players areplaying. The model data and the world are internally represented by amethod called scene graph, and, after the generation of the scene graphof the world, the scene graphs is subjected to rendering. In thisoperation, the rendering is executed on a scene seen from the positionand attitude, given from the player position and attitude determiningmeans 402.

[0085] The rendering may be executed on an unrepresented internal memoryor on a display memory called a frame buffer. For the purpose ofsimplicity, the rendering is assumed to be executed on the unrepresentedinternal memory.

[0086] The image composition unit 404 superimposes the CG, generated bythe CG generation unit 405, with an image obtained by the image pickupunit 302 in the HMD 103. For superimposed display of images, there canbe utilized a method called alpha blending. In case the imagecomposition unit 404 has a pixel output format RGBA including an opacityA (alpha value; 0 A 1) in addition to the intensities of three primarycolors RGB, the image synthesis can be executed utilizing such opacityvalue A.

[0087] As an example, let us consider a case where a pixel on the outputfrom the image pickup unit 302 has RGB values (R₁, G₁, B₁), while acorresponding pixel of the image composition unit 404 has RGB values(R₂, G₂, B₂) and an opacity value A.

[0088] In such case, the corresponding pixel values outputted to thedisplay unit 303 are given by:

(R ₁*(1−A)+R₂ *A, G ₁*(1A)+G ₂ *A, B ₁*(1A)+B ₂ *A).

[0089] Such alpha blending process can also be executed in the CGgeneration unit 405, but separate components are illustrated for ease ofexplanation of the functions.

[0090] The above-described configuration allows the player to feel aheightened feeling of reality in addition to the interestingness of thegame itself. Also the player can easily grasp the proceeding situationof the game, as the superimposed CG are synchronized with the proceedingof the game.

[0091] As the position and attitude information, there is only requiredthe relative relationship between the board 101 and the HMD 103 inposition and attitude, and there is not required the scene setting orthe setting of stage properties or the calibration of the sensor to bemounted on the player, at each installation of the game.

[0092] The present invention is not limited to games but also isapplicable to various field such as education, presentation, simulationor visualization.

[0093] (Second Embodiment)

[0094]FIG. 5 shows an example of the configuration of the imageexperiencing system of a second embodiment, which is different from theembodiment shown in FIG. 4 only in the configuration of the playerposition and attitude determining means 402.

[0095] The player position and attitude determining means 402 iscomposed of a camera 501 fixed to the HMD 103 and a board imagerecognition unit 502. The image of the board 101, in the image taken bythe camera 501, varies depending on the position of the view point 202of the player. Therefore the image taken by the camera 501 is analyzedby the board image recognition unit 502, to determine the position andthe attitude of the view of the player.

[0096] The board image recognition unit recognizes the image of theboard 101, but the image obtained by the camera 501 appears distortedwhen the markers 201 are attached to the game board 101. The positionand attitude of the camera 501 can be determined based on suchdistortion. It is known that the position and attitude of the camera 501can be determined if the markers 201 in at least four points arecorrelated. Thus determined position and attitude of the camera 501 arecorrected based on the difference between the camera position and theposition of the view point of the player to output the position andattitude of the view point 202 of the player.

[0097] In case the image pickup unit 302 is attached to HMD 103, suchimage pickup unit 302 may be used instead of the camera 501 for similareffects.

[0098] Thus, in the present embodiment, the player position and attitudedetermining means is composed of the camera fixed to the see-through HMDand the board image recognition unit, and the board image recognitionunit determines the relative position and attitude between the board andthe view point of the player based on the image of the board taken bythe camera, so that the position and attitude information of the viewpoint of the player can be determined without the position and attitudesensor, whereby the configuration can be simplified.

[0099] (Third Embodiment)

[0100]FIG. 6 shows an example of the configuration of the imageexperiencing system of a third embodiment, which is different from theembodiments shown in FIGS. 4 and 5 only in the configuration of theplayer position and attitude determining means 402.

[0101] The player position and attitude determining means 402 isprovided with both components of the means 402 shown in FIGS. 4 and 5,and additionally with a most likely position and attitude determiningunit 601.

[0102] In general, the output of the position and attitude sensor 104 issusceptible to external perturbations and is rather unstable. Therefore,it is conceivable to use the position and attitude information from theboard image recognition unit 502 as the output of the player positionand attitude determining means, but the board 101 is not necessarilyalways included in the image taking range of the camera, and there mayalso be an element hindering the recognition, such as a hand of theplayer. Therefore, the reliability of the position and attitudeinformation is deteriorated in such situation.

[0103] Therefore, the information from the position and attitude sensor104 is utilized only in such situation. Such configuration allows toobtain the output of the position and attitude information withoutinterruption, and to obtain higher precise position and attitudeinformation while the board 101 is recognized.

[0104]FIG. 7 is a UML activity chart showing the process of the mostlikely position and attitude determining unit.

[0105] At first, the unit awaits the position and attitude informationfrom the position and attitude sensor information processing unit 403and from the board image recognition unit 502. When both data becomeavailable, there is discriminated whether the position and attitudeinformation data from the board image recognition unit 502 are suitable.If suitable, the information from the board image recognition unit 502is used as the output of the player position and attitude determiningmeans 402, but, if not suitable, there is used the information from theposition and attitude sensor information processing unit 403.

[0106] In the foregoing it is assumed that highly precise values areobtained from the board image recognition unit 502 during a properrecognition but unsuitable values are obtained otherwise, but thepresent invention is not limited to such case. A certain image from thecamera 501 may only provide values of low reliability, and in a certainposition and attitude sensor, the reliability may be high only in alimited range and gradually decreases outside such range. Consequentlythe system has to be so designed as to provide most suitable valuesbased on the reliability of the output values of the position andattitude sensor information processing unit 403 and the board imagerecognition unit 502.

[0107] The present embodiment allows to obtain reliable position andattitude information even in case any of the position and attitudeinformation determined by plural methods.

[0108] (Fourth Embodiment)

[0109] The position and attitude sensor is available in various types,but such sensor in many types is associated with a drawback offluctuation of the obtained values. FIG. 8 is a chart showing thedifference dV between the measured value and the true value as afunction of elapsing time in the abscissa. There are experienced a smallfluctuation as indicated by a broken line, and a large shift asindicated by a solid line, and the present embodiment deals with thecase of a large shift as indicated by the solid line.

[0110] In case of a large shift, the difference of two consecutivesamples of dV is small if the sampling is executed with a sufficientlyhigh frequency. Therefore, in a state where the output value of theboard image recognition unit 502 is suitable, dV is calculated assumingsuch value as the true value and −dV is used as the correction value forthe position and attitude sensor information.

[0111] In this manner, a large shift in the value starts from 0 when theoutput value of the board image recognition unit 502 becomes unsuitable,and the position and attitude information released by the playerposition and attitude determining means 402 in such state remainscontinuous, so that the player is relieved from an unpleasant feelingcaused by a sudden shift in the position of the CG image.

[0112] Also in case the unsuitable period of the value from the boardimage recognition unit 502 is sufficiently short, the change in dVresulting from a large shift is small, so that the CG drawing positiondoes not become discontinuous when the output value of the board imagerecognition unit 502 is adopted as the value of the player position andattitude determining means 402.

[0113]FIG. 9 shows an example of the image experiencing system of thefourth embodiment.

[0114] The basic configuration is same as in FIG. 6, except that theoutput of the board image recognition unit 502 is entered into theposition and attitude sensor information processing unit 403 in additionto the most likely position and attitude determining unit 601, and theoutput of the position and attitude sensor information processing unit403 is entered into the board image recognition unit 502 in addition tothe most likely position and attitude determining unit 601. In thefollowing description, however, the input of the output of the positionand attitude sensor information processing unit 403 into the board imagerecognition unit 502 is considered negligible.

[0115]FIG. 10 is a UML activity chart showing the process of theposition and attitude sensor information processing unit 403.

[0116] The information from the position and attitude sensor 104 isprocessed in normal manner to calculate the position and attitudeinformation, and its value is retracted as a variable LastSensorPro. Thevariable LastSensorPro is an object variable which is referred to alsofrom another thread to be explained in the following.

[0117] Subsequently, the calculated position and attitude information isadded with a correction value to obtain a return value which istemporarily retracted. The correction value is also an object variable,of which value is set by another thread to be explained in thefollowing. The return value is a local variable, which is onlytemporarily used for an exclusive execution. Finally, the return valueis returned as the output of the position and attitude sensorinformation processing unit 403.

[0118] The aforementioned correction value, which is from time to timerenewed in response to an output from the board image recognition unit502, is calculated in the following manner.

[0119] At first there is discriminated whether the image recognitioninformation is suitable. If unsuitable, the renewing process is notexecuted. If suitable, the correction value is set by subtracting thevariable LastSensorPro from the position and attitude informationobtained by the image recognition.

[0120] In the foregoing, the input from the position and attitude sensorinformation processing unit 403 to the board image recognition unit 502is considered negligible, but the present invention is not limited tosuch case. In case highly precise information can be obtained from theposition and attitude sensor 104, the correction value can be renewedalso in the board image recognition unit 502 in a similar manner as inthe position and attitude sensor information processing unit. Also theextent of correction on the respective values may be varied depending onthe confidences on such values. For example, in case the difference ofthe confidences is very large, the correction value for the position andattitude determining means at the lower side is so renewed as tosubstantially directly release the output at the higher side, but, incase the difference is not so large, the correction value is so renewedas to execute corrections in small amounts on all the values.

[0121] According to the present embodiment, the renewal of thecorrection value allows to constantly obtain the position and attitudeinformation on the view point of the player, with a high reliability, ina continuous manner, thereby avoiding the unpleasant feeling resultingfrom a sudden shift in the CG drawing position.

[0122] (Fifth Embodiment)

[0123] It is explained in the foregoing that the position and attitudesensor can be provided by the combination of a gyro sensor and anacceleration sensor, in which the gyro sensor detects the attitudeinformation only. Such attitude sensor can be utilized as the positionand attitude sensor if the position is calibrated in advance. Suchcalibration can be dispensed with if there is simultaneously providedposition and attitude determining means consisting of the camera 501 andthe board image recognition unit 502.

[0124] The position and attitude information is basically calculated byimage processing, and, if such information obtained by the imageprocessing is unsuitable, the attitude data alone can be compensated bythe value supplied from the attitude sensor. In case of a board game ora card game, the change in the viewing field resulting from a change inthe attitude of the HMD 103 is considered much larger than thatresulting from a change in the position of the HMD 103, so that thecompensation of the attitude information alone can be consideredsignificantly useful. For this reason, an attitude sensor is fixed, inaddition to the camera 510, to the HMD 103.

[0125] This constitutes the image experiencing system of a fifthembodiment, of which configuration is shown in FIG. 11.

[0126] In comparison with the embodiment shown in FIG. 6, the presentembodiment is different only in the configuration of the player positionand attitude determining means 302. More specifically, the position andattitude sensor 103 is replaced by an attitude sensor 1101, the positionand attitude sensor information processing unit 403 is replaced by anattitude sensor information processing unit 1102, and the most likelyposition and attitude determining unit 601 is replaced by a most likelyattitude determining unit 1103.

[0127] The basic process flow is same as in the third embodiment. Theoutput data of the attitude sensor 1101 are processed by the attitudesensor information processing unit 1102 to provide attitude information.It is to be noted that the position and attitude sensor informationprocessing unit 403 outputs the position information, in addition to theattitude information.

[0128]FIG. 12 is a UML activity chart showing the process of the mostlikely attitude determining unit 1103.

[0129] At first, the unit awaits the attitude information from theattitude sensor information processing unit 1102 and the imagerecognition information from the board image recognition unit 502. Whenboth data become available, there is discriminated whether the imagerecognition information is suitable.

[0130] If suitable, the position information alone therein is set as anobject variable LastIPPos. Then the image recognition information isreturned and the process is terminated.

[0131] If the information is identified not suitable, the attitudesensor information is used as the attitude information, and the variableLastIPPos set in the foregoing is used as the deficient positioninformation. Then the position and attitude information, obtained bycombining both data, is returned and the process is terminated.

[0132] In the foregoing it is assumed that the values of the board imagerecognition unit are more reliable that those of the attitudeinformation processing unit with respect to the attitude information,but the most likely attitude determining unit 1103 may calculate anddetermine the attitude information from both values, depending on thereliabilities thereof.

[0133] In the present embodiment, as the most likely attitudedetermining unit determines the most reliable attitude of the view pointof the player based on the respective output values, utilizing thereliabilities thereof, there can be obtained highly reliable attitudeinformation on the view point of the player, even in case any of theoutput values is unsuitable.

[0134] The present embodiment allows to dispense with the calibration ofthe position and attitude of the view point of the user, which isindispensable in case of employing the attitude sensor only, and toprovide an inexpensive system because an attitude sensor can be employedinstead of the position and attitude sensor.

[0135] (Sixth Embodiment)

[0136] As explained in the foregoing, also the attitude informationobtained from the attitude sensor information processing unit 1102 showsfluctuation. The present embodiment resolves such drawback by a methodsimilar to the fourth embodiment.

[0137]FIG. 13 shows an example of the configuration of the imageexperiencing system of a sixth embodiment, which is same in theconfiguration as in FIG. 11 except that the output of the board imagerecognition unit 502 is supplied not only to the most likely attitudedetermining unit 1103 but also to the attitude sensor informationprocessing unit 1102. However, the attitude sensor informationprocessing unit 1102 receives only the attitude information within theposition and attitude information.

[0138]FIG. 14 is a UML activity chart showing the process of theattitude sensor information processing unit 1102.

[0139] The information from the attitude sensor 1101 is processed in thenormal manner to obtain the attitude information, of which value isretracted as a variable LastSensorDir. The variable LastSensorDir is anobject variable which is referred to also from another thread to beexplained in the following.

[0140] Subsequently, the calculated attitude information is added with acorrection value to obtain a return value which is temporarilyretracted. The correction value is also an object variable, of whichvalue is set by another thread to be explained in the following. Thereturn value is a local variable, which is only temporarily used for anexclusive execution. Finally, the return value is returned as the outputof the attitude sensor information processing unit 1102.

[0141] The aforementioned correction value, which is from time to timerenewed in response to an output from the board image recognition unit502, is calculated in the following manner.

[0142] At first there is discriminated whether the image recognitioninformation is suitable. If unsuitable, the renewing process is notexecuted. If suitable, the correction value is set by subtracting thevariable LastSensorDir from the attitude information obtained by theimage recognition.

[0143] In the foregoing, the input from the attitude sensor informationprocessing unit 1102 to the board image recognition unit 502 isconsidered negligible, but the present invention is not limited to suchcase. In case highly precise information can be obtained from theattitude sensor 1101, the correction value can be renewed also in theboard image recognition unit 502 in a similar manner as in the attitudesensor information processing unit. Also the extent of correction on therespective values may be varied depending on the confidences of suchvalues. For example, in case the difference of the confidences is verylarge, the correction value for the attitude determining means at thelower side, or a portion relating to the attitude determination in theposition and attitude determining means, is so renewed as tosubstantially directly release the output at the higher side, but, incase the difference is not so large, the correction value is so renewedas to execute corrections in small amounts on all the values.

[0144] The present embodiment allows to obtain the position and attitudeinformation on the view point of the player, including the attitudeinformation of a high reliability, in a continuous manner.

[0145] (Seventh Embodiment)

[0146] It is already explained that the board 101, constituting a fieldof the board game, includes certain areas and the players execute thegame by placing, removing or moving pieces in, from or between theseareas. The game management unit 401 grasps the situation of the scene orproceeding of the game, to enable the CG generation unit 405 to generateCG matching such scene or proceeding of the game, whereby the game isfelt more realistic to the players.

[0147] For this purpose, there is provided, for the piece to be operatedby the player, piece operation recognition means for recognizing “whichpiece” is “placed/removed” in or from “which area”.

[0148] It is naturally possible also to employ another item in place forthe piece and to recognize the operation on such item.

[0149]FIG. 15 shows an example of the configuration of the imageexperiencing system of a seventh embodiment.

[0150] A piece operation recognition unit A 1501 is composed of aspecial mark such as a bar code attached to a piece, and a special markrecognition unit A 1502 such as a bar code reader for regonizing thespecial mark. The special mark is to be attached on the piece and istherefore omitted from FIG. 15.

[0151] The special mark is used only for identifying the piece, and cannot only be an ordinary printed mark but can also be based on so-calledRFID system utilizing an IC chip or the like.

[0152] The special mark recognition unit A 1502 may be provided in eacharea on the board 101, or may be provided collectively for plural or allthe areas on the board.

[0153] The data from the special mark recognition unit A 1502 aretransferred to a special mark recognition unit B 1503, and then to apiece operation recognition unit B 1504.

[0154] The special mark recognition unit B 1503 analyzes the informationfrom the special mark recognition unit A 1502 and converts it into adata format required by the piece operation recognition unit B 1504. Incase a special mark recognition unit B 1503 is provided in each area,the information of “which area” can be identified from the special markrecognition unit B 1503 releasing the output and need not be released.However, in case a single special mark recognition unit B 1503 coversplural areas, the information indicating “which area” is outputted forthus covered areas. Also, in case the input from the special markrecognition unit A 1502 is for example a number of 10 digits, such inputis converted, for example by a conversion table, into informationindicating “which area”.

[0155] The piece operation recognition unit B 1504 recognizes “whichpiece” is “placed/removed” in or from “which area”, and transfer theresult of such recognition, as the result obtained by the pieceoperation recognition unit A 1501, to the game management unit 401.

[0156] The game management unit 401 causes the game to proceed, based onthe result of recognition from the piece operation recognition unit A1501. In the actual proceeding of the game, there may be requiredinformation that “which piece” is moved from “which area” to “whicharea”. Such information is judged by the game management unit 401, bycombining information that a piece is “removed from an area j” andinformation that “a piece i is placed in an area k”. In this case, ifthe piece placed in the area j is a piece i, there is judged that “apiece i is moved from an area j to an area k”. The piece in the area jcan be identified as the piece i by managing and referring to thehistory by the game management unit.

[0157]FIG. 16 is a UML activity chart showing the process of the pieceoperation recognition unit.

[0158] The special mark recognition unit A 1501 is provided in each areaon the board 101, whereby a special mark recognition unit i correspondsto the area i. The special mark recognition unit B 1503 returns aspecial mark identifier j when the piece j is placed, and a particularspecial mark identifier Nothing when the piece is removed.

[0159] The piece operation recognition unit A 1501 awaits the input fromthe special mark recognition unit, and outputs a result “a piece isremoved from the area i” or “a piece j is placed in the area i”respectively if the special mark identifier j is Nothing or otherwise.

[0160] The present embodiment allows the game to proceed, based on theactual operations of the players. Since the CG can be generated matchingthe scene or proceeding situation of the game, it is felt as a morerealistic game to the players.

[0161] (Eighth Embodiment)

[0162] A bar code can be used as the special mark identifier,corresponding to claim 10.

[0163] The bar code is widely utilized for example in the field ofdistribution of commodities, and has various features such as easyavailability, high accuracy in recognition, stability in recognition,inexpensiveness etc. Particularly in case of a card game, the bar codecan be printed simultaneously with the printing of the cards. Also aninvisible bar code can be used for attaching the special mark withoutaffecting the design of the cards.

[0164] (Ninth Embodiment)

[0165] An RFID system, or radio frequency identification technologywhich is a non-contact automatic identification technology utilizingradio frequency, can be used as the special mark recognition means.

[0166] A device called tag or transponder is attached to an article, andan ID specific to the tag is read by a reader. In general, the tag iscomposed of a semiconductor circuitry including a control circuit, amemory etc. constructed as a single chip, and an antenna. The readeremits an inquiring electric wave, which is also used as electric energy,so that the tag does not require a battery. In response to the inquiringwave, the tag emits the ID stored in advance in the memory. The readerreads such ID, thereby identifying the article.

[0167] The RFID system is widely employed for example in the ID card orthe like, and has features of easy availability, high accuracy inrecognition, stability in recognition, inexpensiveness etc. If the tagis incorporated inside the piece, it can be recognized without affectingat all the external appearance of the piece. Also the piece and theboard have a larger freedom in designing, since the surface of the pieceneed not be flat and a non-metallic obstacle may be present between thetag and the reader.

[0168] (Tenth Embodiment) The “piece” can be recognized, even withoutthe special mark recognition unit, by an image recognition process onthe image obtained with a camera. The piece recognition can be achievedby the pattern on the card surface in case of a card game, or by theshape of the piece in case of chess or the like, or by the shape of thepiece and the pattern drawn thereon in other games.

[0169] In the following there will be explained an example ofrecognizing a pattern drawn on the surface of a rectangular card, butthe present invention is also applicable to a case of recognizing theshape of the piece, or a case of recognizing the shape of the piece andthe pattern thereon at the same time.

[0170]FIG. 17 shows an example of the configuration of the imageexperiencing system of a tenth embodiment. In comparison with theembodiment shown in FIG. 15, the present embodiment is different only inthe configuration of the piece operation recognition means 1501, whereinthe special mark recognition unit 1502 corresponds to a piecerecognition camera 1701 and the special mark recognition unit 1503corresponds to a piece image recognition unit 1702. The piece operationrecognition unit 1504 remains same.

[0171] In the following there will be explained an example ofrecognizing two patterns shown in FIG. 18, but it is also possible torecognize various complex patterns such as a cartoon or a photograph, byemploying more complex processing.

[0172]FIG. 20 is a UML activity chart showing the process of the pieceimage recognition unit 1702.

[0173] The recognition is executed in two stages, namely the detectionof a frame, and then the detection of a pattern. In case the framecannot be detected, it is judged that the card is not present, and apiece identifier Nothing is returned. The method of frame detection isnot illustrated, but can be achieved, for example, by detecting straightlines by Huffman transformation or the like and judging a frame from thepositional relationship of such lines.

[0174] In case the frame is detected, there is then executed detectionof the pattern. As shown in FIG. 19, the interior of the frame isdivided into four areas, and the color is detected in each area. Variousmethods are available also for the color detection. For example, in caseof detecting white and black only, there is only utilized the luminocityinformation and the average luminocity is determined in the object area,and the area is judged as black or white respectively if such averageluminocity is lower or higher than a predetermined value T_(B).

[0175] The areas are numbered from 1 to 4 as shown in FIG. 19, and, ifthe combination of colors in the areas 1 to 4 areblack-white-white-black or white-black-black-white, a piece identifier 1is returned. If the colors are black-black-white-white,white-white-black-black, black-white-black-white orwhite-black-white-black, a piece identifier 2 is returned. Any othercombination indicates an unexpected card or an erroneous recognition ofthe frame, and an identifier Nothing is returned, indicating the absenceof the card. In case the image recognition is repeated, a same result isoutputted in succession.

[0176] In the course of a card placing or removing operation, there maybe outputted at random a state where a card is placed or removed. In agame which may be hindered by such situation, there is required ameasure for suppressing successive same outputs or for providing anoutput when a same state continues for a predetermined time, but suchmeasure will not be explained.

[0177] (Eleventh Embodiment)

[0178] The camera 501 fixed to the HMD 103 can be used for the piecerecognition. In such case the system can be simplified as the camerafixed to the HMD of the player is used for the piece operationrecognition means.

[0179] If the board 101 can be recognized by the board image recognitionunit 502, the areas provided on the board 101 can be identified. Thepiece operation recognition means can be constituted by recognizing thepieces in such areas.

[0180]FIG. 21 shows an example of the configuration of the imageexperiencing system of an eleventh embodiment. Piece operationrecognition means 1501 is composed of an on-board piece imagerecognition unit 2101 and a piece operation recognition unit 1504, andthe image data to be recognized are entered from the camera 501 whilethe recognition information of the board 101 is entered from the boardimage recognition unit 502. The output information of the board imagerecognition unit 502 indicates the position and attitude of the viewpoint of the player, from which the position of the board on the imagecan be easily calculated.

[0181]FIG. 22 is a UML activity chart showing the process of theon-board piece image recognition unit. The unit receives an image inputfrom the camera 501, and then the position and attitude of the viewpoint of the HMD from the board image recognition unit 502.

[0182] At first the position and attitude of the board 101 on the inputimage is calculated from the position and attitude of the view point ofthe HMD. Then the positions and attitudes of the areas on the inputimage are calculated from the position and attitude information of thepredetermined areas on the board.

[0183] Once the position and attitude of each area are known, the imagein such position is cut out and subjected to image recognition torecognize the piece in each area. Since the information of each areaincludes the attitude information, such information may also be utilizedin the image recognition to improve the accuracy.

[0184] (Twelfth Embodiment)

[0185] In case of recognizing a piece on the board by the camera 501,there may arise a situation where the number of pixels occupied by thepiece on the image becomes smaller, for example depending on thedistance from the camera 501 to the piece or on the attitude of thepiece relative to the camera 501, thereby rendering the recognitiondifficult or requiring a complex configuration of the image recognitionunit for correcting the deformation of the image.

[0186] Therefore, in recognizing “which piece”, the piece is brought toa predetermined specified position relative to the camera 501.

[0187] For example, the piece is brought to a position at a distance of30 cm in front of the camera. For example, in case of a card as shown inFIG. 18, the card is judged exposed to the camera when the frame arrivesat a specified position on the image, and the recognition of the card isexecuted in such position.

[0188] Once the piece is recognized, “which area” can be identified bytracing the piece until it is placed on the board or by recognizing“placing” of any piece by simplifying the image recognition unit in thetenth embodiment. The “removing” can also be recognized in a similarmanner.

[0189] In recognizing “which piece”, the recognition rate can beimproved by positioning the piece at a specified position with respectto a specified camera. Also it is possible to simplify the configurationof the recognition unit.

[0190]FIG. 23 shows an example of the configuration of the imageexperiencing system of a twelfth embodiment. In comparison with theembodiment shown in FIG. 21, the present embodiment is different only inthe configuration of the piece operation recognition means 1501. Theon-board piece image recognition unit 2101 may be same as that in theeleventh embodiment, or may be further simplified since there is onlyrequired judgment that a piece is “placed” or “removed”. The piece imagerecognition unit can be same as that shown in the tenth embodiment. Apiece operation recognition unit 2301, different from the pieceoperation recognition unit 1504, receives the inputs from both theon-board piece image recognition unit 2101 and the piece imagerecognition unit 1702.

[0191]FIGS. 24 and 25 are UML activity charts showing the process of thepiece operation recognition unit 2301. FIG. 24 shows a state ofreceiving information “a piece j is recognized” from the piece imagerecognition unit 1702, and FIG. 25 shows a state of receivinginformation “a piece is placed/removed in an area i” from the on-boardpiece image recognition unit 2101.

[0192] In case information “a piece j is recognized” is received fromthe piece image recognition unit 1702, the information “piece j” isrecorded in an object variable, and is utilized as the information“which piece” when information “piece is placed in which area” isreceived later.

[0193] Also in case information “a piece is placed in an area i” isreceived from the on-board piece image recognition unit 2101, theinformation “piece j” recorded in the object variable is read out, and aresult “a piece j is placed in an area i” is returned.

[0194] Also in case information “a piece is removed from an area i” isreceived from the on-board piece image recognition unit 2101, a result“a piece is removed from an area i” is directly returned.

[0195] The input to the piece image recognition unit 1702 may beexecuted, instead of the camera 501 fixed to the HMD 103, by anexclusive camera such as a separately prepared document camera. Also asimilar effect can be attained by replacing the combination of theexclusive camera and the piece image recognition unit 1702 by thespecial mark recognition unit 1502 and the special mark recognition unit1503, prepared separately.

[0196] (Thirteenth Embodiment)

[0197] The piece can be recognized by the exposure of such piece infront of the camera 501, but such exposure position is not easilyunderstandable for the player. Also if the image experiencing system isso designed as to improve the ease of use by the players, the spatialrange for recognition inevitably becomes wider to result in complicationof the recognition unit or in a loss in the recognition rate.

[0198] It is therefore desired that the player can expose the piece,without doubt or hesitation, to the spatially limited recognition area.This can be achieved by displaying a guide on the display unit 303 ofthe HMD 103, and exposing the piece by the player so as to match thedisplayed guide.

[0199]FIG. 26 shows an example of the configuration of the imageexperiencing system of a thirteenth embodiment. In comparison with theembodiment shown in FIG. 23, the configuration remains same except thatthe piece image recognition unit 1702 is replaced by a piece imagerecognition/guide display instruction unit 2601, from which informationis outputted to the CG generation unit 405.

[0200] The piece image recognition/guide display instruction unit 2601is same in configuration as the piece image recognition unit 1702,except that it outputs a guide display instruction in case theconfidence on the result of recognition is less than a certain level.FIG. 27 shows the difference of the outputs of the piece imagerecognition unit 1702 and the piece image recognition/guide displayinstruction unit 2601. In case the recognition engine is similar to thepiece image recognition unit 1702, there is judged a recognized stateand the result of recognition is outputted if the confidence on therecognition is at least equal to a certain value Th higher than thethreshold value of the piece image recognition unit 1702. A highrecognition rate can be easily realized as the threshold value Th ishigher.

[0201] On the other hand, there is judged a non-recognized state or anon-exposed state of the piece in case the confidence is lower than acertain value Tl. Such value Tl is same as the recognition threshold Thin case of the piece image recognition unit 1702, but is selected lowerin case of the piece image recognition/guide display instruction unit2601. A guide display instruction is given in case the confidence isneither “recognized” nor “not recognized”.

[0202]FIG. 28 is a UML chart showing the process of the piece imagerecognition/guide display instruction unit 2601. If the confidence isselected within a range of 1-0, there stands a relationship 0<Tl<Th<1.After the piece image recognition process, if the confidence on theresult of recognition is lower than Tl, the situation is judgednon-recognized and no action is executed. If the confidence is higherthan Th, the situation is judged recognized and the result ofrecognition is transferred to the piece operation recognition unit 2301.If the situation is neither of the foregoing, a guide displayinstruction is issued. The guide display can be, for example, as shownin FIG. 29.

[0203] In the present embodiment, in case of exposing a piece at aspecified position of a specified camera, a guide for assisting suchexposure is prepared by CG and is displayed in superposition in the HMDof the player, the player can easily place and expose the piece in aspatially appropriate position.

[0204] (Fourteenth Embodiment)

[0205] In a game played by plural players, the event on the board 101,including the display, has to be shared by all the players. This can beachieved logically by sharing a game management unit 401 by all theplayers. Physically, such unit may consist of a single exclusive PC or aspecified exclusive game console including other constituent components,or may be provided in plural game consoles or PCs as in the case of adispersed database.

[0206] Stated differently, for each player, the game console or PC 102assumes a configuration as shown in FIG. 4, and the gate management unit401 also reflects the result of operations executed by other players.

[0207]FIGS. 30A and 30B show examples of the configuration of the imageexperiencing system of a fourteenth embodiment.

[0208] A game console or PC 102 is assigned to each player, and suchconsoles or PCs are mutually connected by a network. The informationflowing in the network is utilized for synchronizing the contents of thegame management units 401 in the game consoles or PCs.

[0209] The piece operation recognition means is provided in each gameconsole or PC 102, with each piece being recognized from plural viewpoints. Ikn such case, it is also possible to exchange the informationof recognition through the network, and to utilize the result ofrecognition of a higher reliability.

[0210] Also FIGS. 31A to 31C show another examples of the configurationof the image experiencing system of the fourteenth embodiment. Gamecontents are contained in a game server on the internet, and the gameconsoles or PCs 102 of the players are connected through the internet.

[0211] The game management unit 401 is composed of local game managementunits 3101 and a game server 3102, the latter being provided in anindependent equipment. The local game management unit 3101 deals withmatters relating only to each player and those requiring feedback toeach local player without delay in time. Also data and programs relatingto the individual game contents are downloaded from the game server 3102through the internet, either at the start of the game or in the courseof execution thereof.

[0212] The present embodiment allows plural players to play on a singleboard, and to display the result of complex operations by the pluralplayers in each HMD based on the view point of each player, therebyenabling experience of a game played by plural players.

[0213] (Fifteenth Embodiment)

[0214] The present embodiment provides a system of executing a game byadapting MR (mixed reality) technology to a card game, thereby combininga real field and a virtual field by CG (computer graphics).

[0215] The card game is already known in various forms, such as poker orblack jack utilizing the playing cards. Recently in fashion is a cardgame utilizing cards in which the cartoon characters are assigned.

[0216] Such card game is played on a play sheet or a game board, byplayers each holding cards. Each card records a cartoon character andits attributes or its specialty skill. The players execute the game byusing these cards, and the game is won or lost by the offensive methodor power and the defensive method or power, which are determined by thecombination of the cards.

[0217] In the following there will be explained the system of thepresent embodiment, applied to a battle-type card game played on aboard, by two players holding the cards.

[0218]FIG. 32 is a conceptual view of the game of the presentembodiment. The two players respectively wear the see-through HMDs, andare positioned across a board, which constitutes the battle space of thegame. The two players play the game by placing the respective cards onthe board or moving the cards placed thereon. In the see-through HMD ofeach player, CG matching the characteristics of each card are displayedon each card.

[0219]FIG. 33 schematically shows the configuration of the system of thepresent embodiment.

[0220] The player wears an HMD 3321, which is provided with a camera3320 and a three-dimensional position and attitude measuring means 3322.The HMD 3321 is connected to a game management unit 3325 while thecamera 3320 and the three-dimensional position and attitude measuringmeans 3322 are connected to a position and attitude grasp unit 3329,both through signal cables.

[0221] The camera 3320 is matched with the view of the player andphotographs the objects observed by the player. The obtained image dataare transferred to the position and attitude grasp unit.

[0222] While the player observes a play board 3326, the image taken bythe camera 3320 contains markers 3331 shown in FIG. 34. The markers 3331are provided in predetermined positions on the play board, and thepositional information of such markers is inputted in advance in theposition and attitude grasp unit 3329. Therefore the area observed bythe player can be estimated from the markers 3321 appearing on the imageof the camera 3320.

[0223] The three-dimensional position and attitude measuring means 3322measures the position and attitude of the HMD worn by the player, andthe measured position and attitude data are supplied to the position andattitude grasp unit 3329.

[0224] Based on the positions of the markers 3331 taken by the camera3320 and the position and attitude data from the three-dimensionalposition and attitude measuring means, the position and attitude graspunit 3329 calculates the range observed by the player.

[0225] Above the play board 3326, there is provided a roof 3327 on whichinstalled is a card recognition camera 3328. The card recognition camera3328 may cover the entire area of the play board 3326, but it is alsopossible to divide the play board into four areas and to place four cardrecognition cameras 3328 respectively corresponding to these dividedareas, or to place card recognition cameras 3328 by a numbercorresponding to that of the areas in which the cards are to be placed.The card recognition camera 3328 constantly watches the play boardduring the game, and the obtained image is transferred to a card readingunit 3324, which identifies the card on the play board 3326, based onthe obtained image.

[0226] In the following there will be given an explanation on themarkers provided on the play board. FIG. 34 is a plan view of the playboard 3326, on which the markers 3331 are provided. Each marker isformed specified color and shape, and the information of such color,shape and position is registered in advance in the position and attitudegrasp unit 3329. By detecting the markers in the image taken by thecamera 3320 and identifying the color and shape, a position in the playboard corresponding to the taken image can be identified. Then, based onthe result of such identification, it is possible to estimate the areaof the board observed by the player.

[0227] In the following there will be given an explanation on guides. Asshown in a plan view of the play board 3326 shown in FIG. 35, the playboard 3326 is provided with guides 3341 in a 5×2 arrangement for theplayer at the front side and also in a 5×2 arrangement for the player atthe rear side. The guide 3341 defines an area in which the card is to beplaced, and a card placed in any other area will be irrelevant from theproceeding of the game. The precision of card recognition can beimproved since the card position is clearly determined by the guide3341, which can be composed, for example, of a recess or a ridgecorresponding to the card size.

[0228] Now the flow of the game will be explained with reference to flowcharts.

[0229] At first reference is made to FIG. 38 for explaining the cardreading unit 3324.

[0230] After the process is started in a step S701, the sequence entersan image capturing phase in a step S702, in which the image of the playboard from the camera 3328 is captured. Then the captured image isidentified in a card identification phase of a step S703.

[0231] The card identification will be explained with reference to FIG.36, which is a plan view of the play board 3326 in a state where threecards are placed on the guides shown in FIG. 35. With a coordinatedescription in which the upper left corner is defined by (0, 0) and thelower right corner by (5, 4), FIG. 36 shows a state where a card ‘G’ isplaced at (5, 1), a card ‘2’ at (4, 3) and a card ‘1’ at (3, 4).

[0232] A step S703 analyzes the camera image, detects the cards placedon the board by the image recognition technology, and identifies thecoordinate value and the kind of each card. The coordinate value and thekind, thus identified, of the cards are retained as card arrangementdata.

[0233] A step S704 compares the present card arrangement data with theprior data. If the comparison shows no change in the arrangement data,the sequence returns to the image capturing step S702. If thearrangement is changed, a step S705 executes a change in the cardarrangement data and the sequence returns to the image capturing stepS702.

[0234] In the following reference is made to FIG. 39 for explaining theposition and attitude grasp unit 3329.

[0235] After the process is started in a step S801, the sequence entersan image capturing phase in a step S802, in which the image from thecamera 3320 attached to the HMD of the player is captured. Then a stepS803 fetches the attitude data from the three-dimensional position andattitude measuring means 3322. A step S804 identifies the markers 3331from the image fetched in the step S802, thereby estimating the viewpoint of the player. Then a step S805 determines the more exact positionand attitude of the view point of the player, based on the attitudeinformation obtained in S803 and the estimated information in S804.Thereafter the sequence returns to S802.

[0236] In the following, reference is made to FIG. 40 for explaining thegame management unit 3325.

[0237] After the process is started in a step S901, a step S902 waitsfor any instruction from the player on the proceeding of the game. Ifany event arrives, the sequence proceeds to a step S903 for identifyingthe kind of the event. If the event is a signal for advancing to a nextphase, the sequence proceeds to a step S904, but, if otherwise, thesequence returns to S902 to wait for a next event.

[0238] The signal for advancing to a next phase is generated byidentifying an operation inducing a phase advancement. Such operationinducing a phase advancement may be recognized by various methods.

[0239] The game advancement can be judged, for example, by recognitionof a voice of the player by a voice recognition unit as shown in FIG.37, or by image recognition of a card exposed by the player in a largeimage size to the camera 3320 attached to the HMD worn by the player, orby image recognition of a card placement in a specified position on theboard in the image obtained from the card recognition camera 3328.

[0240] A step S904 reads the card arrangement data determined by thecard reading unit 3324. Then a step S905 fetches, from the cardarrangement data, the data of the cards relating to the current phase,and calculates the offensive character and the offensive characteristics(offensive power and method) of the offensive side, based on thearrangement and combination of the cards. Then a step S906 fetches, fromthe card arrangement data as in the step S905, the data of the cardsrelating to the current phase, and calculates the defensive characterand the defensive characteristics (defensive power and method) of thedefensive side, based on the arrangement and combination of the cards.The calculation of the offensive and defensive characteristics in thesteps S905 and S906 is executed according to the rules of the game.

[0241] Then a step S907 calculates the result of battle according to thecombination of characters of the offensive and defensive sides, andgenerates a battle scene matching such result. Then a step S908 acquiresthe view point of the player derived by the position and attitude graspunit 3329, then a step S909 generates CG of the battle scene seen fromthe view point of the player, and a step S910 synthesizes the imageobtained from the camera 3320 with the CG generated in S909. Then thereal field and the virtual field of the CG are superimposed by the MRtechnology in the HMD worn by the player, thereby displaying a virtualCG character on the card.

[0242] However, such image synthesis is only required in case of a videosee-through HMD, and not required in case of an optical see-through HMD.

[0243] As explained in the foregoing, the present invention allows tocombine the real world with the virtual CG world by displaying thevirtual CG corresponding to the viewing point, thereby realizing muchhigher excitement in a game utilizing a play board.

[0244] In case the card recognition camera 3328 cannot be installed insatisfactory manner, it is also possible, in order to improve thereading accuracy of the card recognition camera 3328, to adopt a methodin which the player exposes a card in front of the HMD 3321 worn by theplayer to identify the kind of the card by the camera 3320, before suchcard is placed. Also in order to improve the reading accuracy of thecard recognition camera 3328, it is possible to position such camera3328 in a specified position and to identify the kind of a card by suchcamera 3328 before such card is placed.

[0245] In the foregoing embodiment, a card is employed as the item ofthe game, but there may also be employed other items such as a piece.

[0246] Also the board may have a three-dimensionally stepped structureand the character may be displayed in a position corresponding to theheight of such stepped structure. In such case, a model of thethree-dimensionally stepped structure of the play board is registered inadvance, and the position of the synthesized CG is controlled based onthe three-dimensional structural information corresponding to theposition of the view point.

[0247] Also for identifying the card on the play board, the foregoingembodiment analyzes the image of the camera 3328 and recognizes thepattern on the card by the image recognition technology, but it is alsopossible to attach a bar code to the card and to identify the card witha bar code reader.

[0248] (Other Embodiments)

[0249] The present invention also includes a case of supplying acomputer of an apparatus or a system, connected with various devices soas to operate such devices for realizing the functions of theaforementioned embodiments, with program codes of a software realizingthe functions of the aforementioned embodiments and causing a computer(or CPU or MPU) of such apparatus or system to execute the program codesthereby operating such devices and realizing the functions of theaforementioned embodiments.

[0250] In such case, the program codes themselves of such softwarerealize the functions of the aforementioned embodiments, and the programcodes themselves, and means for supplying the computer with such programcodes, for example a memory medium storing such program codes,constitute the present invention.

[0251] The memory medium for supplying such program codes can be, forexample, a floppy disk, a hard disk, an optical disk, a magnetoopticaldisk, a CD-ROM, a magnetic tape, a non-volatile memory card or a ROM.

[0252] The present invention naturally includes not only a case wherethe functions of the aforementioned embodiments are realized by theexecution of the supplied program codes by the computer but also a casewhere the functions of the aforementioned embodiments are realized bythe cooperation of such program codes with an OS (operating system) oranother application software or the like functioning on the computer.

[0253] The present invention further includes a case where the suppliedprogram codes are once stored in a memory provided in a functionexpansion board of the computer or a function expansion unit connectedto the computer and a CPU or the like provided on such functionexpansion board or function expansion unit executes all the processes ora part thereof under the instructions of such program codes.

What is claimed is:
 1. An image experiencing system for a game whichproceeds by placing an item on a game board, the system comprising:player position and attitude determining means for determining theposition and attitude information of a view point of a player;generation means for generating computer graphics corresponding to theitem on the game board, based on the position and attitude informationof the view point of said player; and a head mounted display fordisplaying said generated computer graphics in superposition with animage of the real world.
 2. A system according to claim 1, wherein theposition and attitude information of said player is informationindicating a relative position of the view point of said player relativeto said game board.
 3. A system according to claim 1, furthercomprising: measurement means for measuring attitude information of saidplayer; wherein said player position and attitude determining meansdetermines the position and attitude information of the view point ofsaid player from the attitude information and pre-calibrated positioninformation of said player.
 4. A system according to claim 1, furthercomprising: a camera fixed to said head mounted display; wherein saidplayer position and attitude determining means analyzes the image ofsaid camera and executes image recognition thereon, thereby obtainingthe position and attitude information of the view point of said player.5. A system according to claim 1, further comprising: a position andattitude sensor for measuring the position and attitude of the player;and a camera fixed to said head mounted display; wherein said playerposition and attitude determining means includes a first position andattitude determining unit for determining the position and attitudeinformation of the view point of the player from the output of saidposition and attitude sensor, and a second position and attitudedetermining unit for determining the position and attitude informationof the view point of the player from an image taken by said camera, andthe position and attitude information of the view point of said playeris determined according to the reliability of the respective outputvalues of said first and second position and attitude determining units.6. A system according to claim 1, further comprising: an attitude sensorfor measuring the attitude of the player; and a camera fixed to saidhead mounted display; wherein said player position and attitudedetermining means includes a first position and attitude determiningunit for determining the position and attitude information of the viewpoint of the player from the output of said attitude sensor, and asecond position and attitude determining unit for determining theposition and attitude information of the view point of the player froman image taken by said camera, and the position and attitude informationof the view point of said player is determined according to thereliability of the respective output values of said first and secondposition and attitude determining units.
 7. A system according to claim5 or 6, wherein a correction value is determined from the output valueof said first or second position and attitude determining unit, and saidcorrection value is used for correcting the output value of said firstor second position and attitude determining unit.
 8. A system accordingto claim 1, further comprising item operation recognition means forrecognizing a change in the item on said game board.
 9. A systemaccording to claim 8, wherein said item operation recognition meansrecognizes a special mark identifier attached to the item.
 10. A systemaccording to claim 9, wherein a visible or invisible bar code is used asthe special mark identifier.
 11. A system according to claim 9, whereina RFID transponder is used as the special mark identifier.
 12. A systemaccording to claim 8, wherein said item operation recognition meansrecognizes a shape of the item and/or a pattern on the item by imagerecognition.
 13. A system according to claim 12, wherein an image takenby a camera fixed to said head mounted display is used.
 14. A systemaccording to claim 12, wherein said item operation recognition meansrecognizes a placement of an item in a specified position of a specifiedcamera, thereby recognizing an item to be changed.
 15. A systemaccording to claim 14, wherein, in placing an item at said specifiedposition of said specified camera, a guide facilitating the placement ofthe item is prepared by computer graphics and is displayed in the headmounted display of said player.
 16. A system according to claim 1,wherein plural players play on a single game board, and the result ofcomplex operations by the plural players is displayed in each headmounted display in the view point of each player.
 17. An informationprocessing method for a game which proceeds by placing an item on a gameboard, the method comprising steps of: entering position and attitudeinformation of a view point of a player; generating computer graphicscorresponding to the item on the game board, based on the position andattitude information of the view point of said player; and displayingsaid generated computer graphics in superposition with an image of thereal world in a head mounted display worn by the player.
 18. A methodaccording to claim 17, wherein the position and attitude information ofsaid player is information indicating a relative position of the viewpoint of said player relative to said game board.
 19. A program forcontrolling an information processing apparatus thereby executing theinformation processing according to claim
 17. 20. A recording mediumstoring the program according to claim
 19. 21. An image experiencingsystem for a game which proceeds by placing an item on a game boardbearing plural marks, the system comprising: a position and attitudegrasp unit for recognizing the kind of plural items placed on said gameboard and the position of said items; game management means for managingthe proceeding of the game, based on the kinds of said plural items andthe positions thereof; and generation means for generating computergraphics of a game scene corresponding to the kinds of the items of saidplural items and the positions thereof.
 22. A system according to claim21, wherein: said game is a battle type game; said system furthercomprises determination means for determining the characteristics of acharacter of an own side based on the combination of the own side; andsaid generation means generates said computer graphics based saiddetermined characteristics of the character of the own side.
 23. Asystem according to claim 21, wherein said game management means managesthe proceeding of the game by a combination of the characteristics ofthe characters of the own side and the opponent side.
 24. A systemaccording to claim 21, wherein said game board has a three-dimensionalstructure, and said computer graphics are displayed in the head mounteddisplay worn by the player, based on a model of the three-dimensionalstructure of said game board.
 25. An information processing method for agame which proceeds by placing an item on a game board bearing pluralmarks, the method comprising steps of: entering an image from an imagepickup unit in a head mounted display worn by a player; detecting a markin said image thereby determining the position and attitude of a viewpoint of said player on said game board; identifying the item on saidgame board and generating computer graphics indicating a game scenebased on the result of said identification; and displaying said computergraphics, based on the position and attitude of said view point in saidhead mounted display.
 26. A method according to claim 25, furthercomprising a step of: entering position and attitude information of theplayer from a three-dimensional position and attitude measuring unit formeasuring the position and attitude of said player; wherein the positionand attitude of said view point is obtained from said position andattitude information and from said detected mark.
 27. A method accordingto claim 25, wherein the item on said game board is identified from animage from an image pickup unit for taking the image on said game board.28. A method according to claim 25, further comprising a step ofrecognizing an instruction from the player for advancing the game.
 29. Aprogram for realizing an information processing method for a game whichproceeds by placing an item on a game board bearing plural marks, themethod comprising steps of: entering an image from an image pickup unitin a head mounted display worn by a player; detecting a mark in saidimage thereby determining the position and attitude of a view point ofsaid player on said game board; identifying the item on said game boardand generating computer graphics indicating a game scene based on theresult of said identification; and displaying said computer graphics,based on the position and attitude of said view point in said headmounted display.